The present application relates to panoramic or semi-panoramic 3D scenes and their capturing.
It is widely accepted that concept of immersive media is one of the most promising market segments of future technology. One feature of immersive media is panoramic imaging using large cylindrically or spherically curved screens, often in combination with multi-projection systems providing ultra-high resolution by exact juxtaposition and blending of multiple projector images[1][2][3][4][5]. Being lost in niche markets like theme parks for a long time, these applications are now migrating into new market segments like event and exhibition technology, training centers or even entertainment. Typical applications are dome projections (e.g. in planetariums), giant screen cinemas (e.g. re-opening of digital Cinerama theatres) or immersive 180° or 360° surround video (e.g. simulation and training centers) [6][7][8]. In future, they may even address immersive viewing in new types of cinema theatres or other public venues, or, to its end, in immersive home entertainment.
In February 2010, the Fraunhofer Heinrich-Hertz-Institute (HHI) in Berlin, Germany, has opened its ‘Tomorrow's Immersive Media Experience Laboratory (TiME Lab)’, an experimental platform for immersive media and related content creation. The TiME Lab uses up to 14 HD projectors for panoramic 2D and 3D projection at a cylindrical 180° screen with a resolution of 7 k×2 k as well as a ‘Wave Field Synthesis (WFS)’ sound system with 128 loudspeakers [9].
Apart from multi-projection, a further main challenge of panoramic imaging is to create live footage supporting these special video formats in combination with ultra-high resolution. One solution is, in analogy to multi-projection, to use multiple cameras where the single cameras look into different directions such the resulting images can be stitched seamlessly to large panoramic views. The technology of such omni-directional camera systems has a long tradition. First systems that use multiple cameras and mirrors to achieve full surround capture with high image resolution have already been used in the 60s by Ub Iwerks for Disney theme park productions [10]. Since then many further mirror-based system approaches have been proposed (e.g. [11]). Other approaches place a hyperboloid mirror in front of a single camera to capture panoramic views [12][13]. Today, the advances and ongoing miniaturization of digital video cameras enables more compact systems and several commercial companies offer omni-directional cameras for a wide range of applications [14][15][16][17][18][19][20][21]. Good overviews about different approaches on panoramic imaging are given in [22][23].
The term “3D panoramic video” is often used for 360° viewing capability in a 2D panorama. However, in this context, two video panoramas of the same scene but with different perspective are considered, one for the left and one for the right eye in order to allow stereoscopic 3D. Although the concept of omni-directional cameras for capturing 2D video panoramas is well understood and a lot of efficient systems are already available, capturing of 3D video panoramas is still a challenge and a partly unsolved problem.
In the following, some review of omni-directional imaging and panoramic 2D video is provided.
As known from projective geometry, the optimal multi-camera arrangement for capturing panoramic videos necessitates that the focal points of all camera views or cameras 2 coincide in a common point 1 (see FIG. 1A) [22][23]. In case of capturing static 2D panoramas, this condition is usually achieved by rotating a single camera at a tripod with a revolving camera head. For video, however, this approach is impractical due to the need of using multiple cameras on one hand and the physical dimensions of each camera on the other hand. Hence, special mirror rigs are often used to achieve the condition from FIG. 1A. A related mirror rig for implementing the arrangement of FIG. 1A is shown in FIG. 1B. To avoid mechanical complexity, many commercial solutions, however, capture video panoramas with the star-like approach from FIG. 1C, an implementation of which is shown in FIG. 1D [15][18][21]. In this case the focal points 3 of all cameras 2 are located on a common circle 4, while the optical axes 5 are perpendicular to the arc 4. This approach works reasonably well as long as only far distant objects appear in the scene. However, the existence of a non-zero parallax angle does not allow seamless stitching in case of close objects in the overlap area (see FIG. 1E).
As said, a suitable approximation of the optimal solution from FIG. 1A can be achieved by using special mirror-rigs as exemplarily shown in FIG. 1B. If all cameras 2 and mirrors 6 are arranged correctly, it is possible to superimpose the virtual images of all focal points in one common central point 1 behind the mirrors 6. Since the first applications in the 60s, many further system approaches have been proposed and have made a lot of progress, last but not least, due to the advent of digital TV and cinema cameras [10][22][23].
In particular, an advanced version of the system shown in FIG. 1B has recently been presented by Fraunhofer HHI. The so-called OMNICAM is a scalable system, which can be equipped with up to 12 HD cameras for 360° shooting. In its implementation shown FIG. 1B, it uses 6 HD cameras 2 suitable to shoot 180° panoramas. The six cameras 2 generate tiles of 1080×1920 pixels each, which can subsequently be stitched to one large panorama with a final resolution of 6984×1920 for 180°. As the cameras 2 are used in portrait format, the vertical field-of-view is about 60°, a feature that is extremely useful for immersive media.
A special property of the OMNICAM of FIG. 1B is its very accurate calibration capabilities shown in FIGS. 2A and 2B. This illustration depicts a horizontal section through the mirror pyramid 6 at the plane where the optical axes 5 intersect the mirror surfaces 6 and, with it, how the virtual images 7 of the focal points are located behind the mirrors 6. Note that the cameras look from bottom upwards and that the mirrors 6 deflect the optical axes horizontally in radial direction (see also FIG. 1B).
In a first step the rig is calibrated such that all virtual images 7 of the focal points coincide in the centre C of the mirror pyramid (see FIG. 2A). This initial state refers to the optimal camera arrangement from FIG. 1A. It is obtained by very precise optical measurements in the laboratory or by a special calibration process on set.
Although this initial and optimal state allows a parallax-free stitching for scenes with a depth range from zero to infinity, it is not really suitable under real working conditions. If all cameras 2 have a common focal point in the center C of the mirror pyramid 6, there would be no overlap between the different tiles due to a hard cut at the mirror edges 8. Hence, there is no possibility to blend pixels between adjacent image tiles. In former applications like theme park productions this drawback has been concealed by segmented projection screens.
However, this is not acceptable any longer for seamless projection of video panoramas in future immersive media applications. Hence, at least some slight overlap between adjacent image tiles is needed. In order to obtain overlaps, the virtual image portion 7 of the focal points of the cameras have to be moved symmetrically by precise actuators out of the center C in radial direction (FIG. 2B). By this off-center adjustment of the focal points, it becomes possible to regulate a scene-adaptive trade-off between sufficient overlaps for blending and parallax-free stitching, an outstanding feature that is not provided by other omni-directional camera systems that are available at the market. In practice, the OMNICAM is usually operated with a radial-shift of about 5 mm, resulting in a blending area of about 10 pixels and a parallax-free stitching of scenes with a depth range from 2 m to infinity. FIGS. 3A to 3D shows an example of intermediate and final results of consecutive steps of a whole OMNICAM processing for a sports production. FIG. 3A) shows the original camera views arranged side by side in their order along the lateral angle direction; FIG. 3B) shows the views after a geometrical correction and warping; FIG. 3C) shows the views after photometrical correction, color matching and blending, i.e. stitched together; and FIG. 3D) shows in a final cut-out of panoramic view created by cropping the views having been stitched together.
Next, possible extensions to Omni-Stereo Imaging and Panoramic 3D video and the problems involved therewith are discussed.
In principle, the above considerations can also be extended towards omni-directional recording of 3D panoramas. However, in the 3D case the situation is much more sophisticated. The main challenge is to solve a fundamental conflict between two competing requirements. On one hand, as in 2D, panoramic 3D imaging also necessitates a parallax-free stitching of the left- and right-eye panoramas. On the other hand, significant parallaxes are needed between the two stereo panoramas to obtain an adequate stereo impression.
Known solutions from literature that solve this problem are mainly suited for static scenes. The capture of static omni-stereo panoramas has already been investigated since more than 15 years. A nice overview on the major principles can be found in [24]. As already mentioned in the previous section, the optimal solution for static 2D panoramas is to rotate a single camera around its focal point (see FIG. 4A). As shown in FIG. 4B, the straight forward extension to 3D is a rotation of a stereo camera 2′ around the center 9 of its baseline B.
From literature, this concept is also known as concentric mosaics, a special version of the plenoptic function [25]. Unfortunately, it is not that easy to apply this solution to the acquisition of 3D video panoramas, especially not for the star-like approach from FIG. 1C. FIG. 5 shows a corresponding star-like arrangement for stereo cameras. As it can be seen from this drawing, the inter-axial distance B of one stereo sub-system is much smaller than the inter-axial distance S between adjacent panorama cameras of same stereo channel (either left-eye or right-eye). This situation perfectly explains the above mentioned conflict between unwanted parallax errors for the stitching process and the necessitated parallax for stereo reproduction. Imagine that the stereo baseline B is well adapted to the near and far objects in the scene such that the resulting parallaxes between the left and right views produce a clearly visible and comfortable stereo effect. Hence, as the inter-axial distance S between adjacent left-eye (or right-eye) panorama cameras is in any case larger than B (or in best case equal to B), the parallax error that appears while stitching the left- and right-eye panoramas is larger than (or equal to) the stereo parallax, if the same near and far objects are also present in the overlap area. Or in other words, if one wants to avoid visible parallax errors while stitching, the stereo effect is lost. The only situation where it works is given by a scene that has enough depth in the single stereo segments but remains flat in the stitching areas. Obviously, such a situation is difficult to control under real shooting conditions.
The optimal solution from FIG. 4B is even more difficult to achieve with real video cameras. The concept of concentric mosaics necessitates that the stereo camera 2′ is rotated in very small angular increments respecting the plenoptic sampling theorem [26]. In the extreme case, the stereo rig even consists of vertical line cameras only and the rotation scans the stereo panorama column by column. Hence, in case of stereo panoramas with ultra-high resolution of several thousand pixels, the angular increment is significantly lower than one degree. It is self-evident that such a set-up cannot be realized with multiple video cameras for 3D video content.
Thus, it is an objective of the present invention to provide a scheme for capturing panoramic or semi-panoramic 3D scenes, which is able to provide high quality 3D scene results at reasonable efforts.